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© Scheduling method and apparatus Idr a communication network. 

@ Method and apparatus for avoiding receive 
livelock and transmit starvation, and for minimizing 
packet k>ss and latency In a communk:alion network 
station. The inventkm uses a combination of pro- 
cessing threads, poHing and, in a preferred embodi- 
ment a Judk:ious use of intemjpts. to anocate the 
use of processing resources fairty among competing 
functkms. Real time processing threads are struc- 
tured to execute for a preselected maximum time 
intend, tweed on numbers of units processed by 
each thread, and then to yield control to a thread 
scheduler, whteh selects and Invokes a new thread 
for executkxt Wori( to be done bi the various 
threads is determined either by pofling or by an 
intem4>t system, and then posted to the threads for 
execution. If interrupts are used, kiterrupt servtee 
routines perform only the minimal processing need- 
ed to recognize an intenrupt, other interrupt senddng 
functk)ns bolvg deferred for execution ki a process- 
ing thread. The thread scheduler operates on a 
round-rot>in t>asis and also selects from at least one 
general purpose processing thread for functions that 
are less time critical, the general purpose tfiread 
being structured to operate for a selected minimum 
time, if needed, in a non-preemptable mode, and 
thereafter in a preemptable mode. 
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BACKGROUND OF THE INVErfOON 

This invention relates generally to the efficient 
design and operation of conununication networks 
and, more particularly, to scheduling mechanisms 
for use in stations connected to a communication 
networic. A network consists of a communication 
channel, and a numt>er of stations connected to the 
channel. Information is transmitted over the channel 
in data frames or packets, the packets having 
headers that permit each station to detenmlne the 
destination of tfie data. 

Each station has to perform a numt>er of tasks 
and sutitasks related to sending and receiving data 
packets. In general, these tasks and subtasks may 
overlap In time, arKJ the performance of any one of 
them may be dependent on the completion of 
anottier. Moreover, the functions performed in a 
station, whether dependent or Independent typh 
cany have relative pdodties. Since a station may 
comprise only one processor ftx- performing all off 
the station tasks and subtasks, scheduling these 
tasks and subtasks at the station is a critical aspect 
of network operations. Scheduling in the end4ys- 
tem has to meet the requirements expected by fho 
network: the network protocols, the rules for acces- 
sing the channel, and so fbrtti, prinnarily In tenns of 
throughput and latency. 

As computer networks have devetoped, various 
approaches have been used in ttie chotee of com- 
munication medium, networic topok)gy, message 
format, and protocols for channel access. Some of 
these approaches have emerged as standards, and 
a model for network architectures has been pro- 
posed and widely accepted. It is known as tfie 
International Standards Organization (ISO) Open 
Systems Interconnection (OSI) reference model. 
The OSI reference model is not itself a networic 
architecture. Rattier It specifies a hierarchy of pro- 
tocol layers and defines the function of each layer 
in the network. Each layer in one computer of the 
networic carries on a conversation witti the cor- 
responding layer in anottier computer witfi whkii 
communication Is taking place. In accordance witti 
a protocol defining the rules of ttiis communfeation. 
In reality, information is transferred down from lay- 
er to layer in one computer, ttien ttuough ttie 
cfiannel medium and back up ttie successive lay- 
ers of ttie ottier computer. However, ffor purposes 
of design of the various layers and understanding 
their functions, it Is easier to consider each of ttie 
layers as communicating witti its counterpart at ttie 
same level on the remote machine. However, of 
interest in the present Invention are ttie data ftow- 
ing from one layer to the next on the same ma- 
chine, and the scheduling of activities of each of 
tfiese layers In the same machine. 



The towest layer defined by the OSI model is 
called the physical layer, and is concerned with 
transmitting raw data bits over the communication 
channel, and making sure that the data tuts are 
5 received wittiout error. Design of ttie physical layer 
involves issues of electrical, mechanical or optical 
engineering, depending on the medium used for 
ttie communication channel. The layer next to ttie 
physical layer is called the data Knk layer. The 

10 main task of ttie data Gnk layer is to transform ttie 
physical layer, whteh Interfaces directiy witti ttie 
channel medium, into a communication 6nk that 
provkJes communk:ation servtees to ttie next layer 
above, known as the networic layer. The data link 

fs layer periorms such functions as structuring data 
into packets or frames, and attaching control in- 
formation to ttie packets or frames, such as check- 
sums for error detection, and packet numbers. 
Afthough ttie data \Wk layer is primarily In- 

20 dependent of the natore of the physical transmis- 
sion medium, certain aspects of the data link layer 
function are more dependent on ttie transmlsston 
medium. For ttiis reason, the data tink layer in 
some network ardutedures is divided into two sut>- 

2S layers: a k)gtoal 6nk control sublayer, whteh per- 
forms an medium-independent functions of tfie data 
link layer, and a media access control (MAC) layer. 
This layer, or sublayer, determines which station 
sfKxikl get access to tfie communication ctiannei 

30 when there Is competition for it The functions of 
the MAC layer are more ikely to bo dependent on 
tfie nature of tfie transmission medium. 

From this t>ackground, it will be appreciated 
that each station must perform a number of house- 
as keeping tasks on a continuing t)asis to ensure 
efficient operation of the station and the network. 
Packets of data received from the network have to 
be promptfy processed to avoid or minimize toss of 
received packets. Processing includes not only de- 

40 fvering tfie packets to a station memory, but trans- 
ferring tfie packets from tfie kiwest protocol layer 
tttfough various higher layers. Packets of data for 
transmission onto the network have to be pro- 
cessed also, i.e. transferred through the protocol 

45 layers and eventually transmitted onto ttie networic 
Processing of data packets may also include exam- 
ining and modifying packet headers. In addition, 
there may be a number of other types of functions 
that the station tias to perfonm, depending on the 

50 activities for whk:h ttie station Is designed. For 
example, a station may be a tile server for tfie 
network, handEng mass storage devices and to 
which other stations have access ttuough tfie file 
server. O tfie station may be a terminal or work- 

65 station through which a user accesses the network. 
In summary, each station has to process receive 
packets and transmit packets, and may have to 
perform a numt)er of other tasks. 
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Some of the station functions are typically per- 
formed in a station adapter, a device ttiat provides 
an interface between the network communication 
~ channel and a host system bus of the station. The 
adapter contains a packet buffer memory In whteh 
receive packets are temporan'ly stored before 
transfer to a host system memory, also connected 
to the bus, and in whk:h transmit packets are 
temporarily stored after retrieval from the host sys- 
tem memory and before being transmitted onto the 
network. The adapter may have Hs own processor, 
or may use a hardware state machine to handle 
some of Its functfons. Other statkm functions may 
be performed by the host processor Itself. Regant- 
less of where at the station the various processing 
functk)ns are performed, there is almost always a 
problem of scheduling these functions In an appro- 
priately efficient way. The problem is simply a 
matter of how to alkx^ate processing resources 
fairly. If a statkm were to be provided with abun- 
dant processing resources, there wouki by no sig- 
nifk:ant scheduling problem, rkxmally. however, 
there is a cost constraint that keeps processing 
resources small, and the schedufing problem usu- 
ally reduces to a question of alk>cating a single 
processor to handle multiple tasks as efficiently as 
possible. 

Typk^ally, scheduling mechanisms in a network 
statkm processor or adapter processor are interrupt 
driven, whteh simply means that the processor is 
intenrupted from time to time by events as they 
transpire. For exampto, when a receive packet 
must be read from the network conununicatkm 
channel, a hardware intenrupt is generated and 
preempts whatever is cunrentiy taking place, in 
favor of the usually mofo Important receive packet 
processing. An interrupted processor must save the 
context of Its current processing task, detenmine 
the cause of the kiternupl, switch to a new task, and 
later return to the interrupted task after restoring 
the context at the tinrte of the interrupt. ScheduGng 
Is often more complex In a host processor, which. 
In additton to corrununteation functions, has to at- 
tend to such activities as disk Input/butput. file 
servk:e, and appKcation protocol processing. 

An interrupt driven scheduling mechanism has 
assigned interrupt prknities, whteh are usually 
fixed. In most systems, the highest priority is ac- 
corded to receiving packets at the bwest network 
protocol layer. Then, the network protocol layers at 
the next higher levels get prkMity to process the 
received packet. When these tasks give up control, 
the k>w6st layers of packet transmlsskm activity get 
priority. Some of these packet reception and trans- 
mission tasks may tie non-interruptable, which is to 
say they must run to some point of completion or 
partial completion before another task obtains con- 
trol. Often, they run to completion, which involves 



processing multiple packets received in a txjrst 

TTiere are four related problems that shouki be 
avokled or minimized in any network station sched- 
uling mechanism: receive livelock, transmit starva- 
5 tion, unacceptat>le packet loss, and unacceptat>le 
latency for delivery of a first packet in bng tHirst of 
received packets. Receive livekKk is a conditton 
that arises when a system is so preoccupied with 
fielding receive interrupts tfiat it is unable to conrv* 

10 plete the process to deliver the packets to the 
appGcatkm, whtoh Is the final redpieni Stated arv 
otfier way, when the processing and delivery of 
receive packets takes k>nger than the time between 
receipt of successive packets, then tfte processor 

rs can only complete processing packets at kywer 
network layers, and does not make suffk^ient 
progress at the higher layer. Receive livelock usu- 
ally results in tiie next mentioned prot>lem. transrr^ 
starvation. A typical scheduling approach provides 

20 receive processing higher prkxity, alkiws receives 
to Intenrupt ether activity, and alkms receive pro- 
cessing to run to corrtpletion before starting ai>- 
other task. In a system emptying this approach, 
progress in preceding transmit packets is assured 

25 only when the packet reception process Is conrv- 
plete. Thus, in a receive fmekxk comfition, or even 
when tfm processor is only relatively txjsy process- 
ing receive packets, the tasks relating to transmits, 
and also to higher layer processing, will be starved 

30 Of processing resources. 

bitenupt driven, non-preemptable processing of 
receives at the kwest network layers also adds 
latency to delivery of packets to thefa- ultimate 
destination, and can introduce excessive packet 

as loss. Latency to deliver the packet to tha applica< 
tion in interrupt driven systems arises because of 
tfie processing overhead needed to servk» an iiv 
tenrupt and to save tfie context of processing al- 
progress, bi addition, because of priority 

40 for tills faitenupt, tiie h^her layers do not make 
timely progress as tttey do not get a fair share of 
processing resources, and latency to defiver the 
packet furttier increases. Packet toss results when 
receive Ivelock Is advanced to tiie state tfiat no 

45 progress Is being made in processing packets al- 
ready received, and newly aniving packets have to 
be dropped for lack of buffers to queue these 
packets to ttie next higher layer. 

It will be appreciated from ttie foregoing tfiat 

so ttiere is still a slgniftoant need for improvement in 
scheduling mechanisms for communication func- 
tions of a network station operating system. Sched- 
uling mechanisms are needed botii in tfie host 
system processing unit and in ttie processor of ttie 

65 adapter tfiat connects tfie host to tfie network. In 
particular, what is needed is a scheduling approach 
that addresses all of the problems discussed 
at>ove. The present invention satisfies this need. 
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SUMMARY OF THE INVENTION 

The present Invention resides in a novel sched- 
uling method and apparatus for use In stations of a 
communication network. The invention practically 
eliminates receive Gvelock and transmit starvation, 
improves performance in terms of packet k>ss and 
latency and induces a greater degree of fairness in 
allocating processing resources to various tasks. 

The invention in its broad form resides in a 
method for scheduling multiple processing tasks, 
as generally recited in daim 1. The invention also 
resteles in apparatus for scheduling multiple pro- 
cessing tasks as t)roadly recited in daim 7. As 
desait)ed hereinafter in greater detail, a preferred 
nteihod of the Invention comprises ttie steps of 
dividing processing tasks into processing threads, 
each of whk:h is stnictured to execute for a limited 
time before being subject to preemption by anotfier 
processing thread; repeatedly determining when 
woric Is to be done by any of the processing 
threads; repeatedly posting any woric to be done to 
appropriate ones of the processing threads; select- 
k)Q a processing thread that has woric to be done; 
invoking the selected processing thread; and ex- 
ecuting ttie selected and Invoked processing 
thread. Most of the threads are categorized as real 
time threads, whk:h execute for up to the selected 
maximum time before yiekfing control. A measure 
of the maximum execution .time In a thread is 
obtained by counting up to a selected number of 
units processed by the thread. The mettiod contin- 
ues by repeating the selecting, invoking and ex- 
ecuting steps for ottier processing threads. 

Tfie maximum numt>er of units processed tjy a 
real time thread each time it Is invoked Is referred 
to as the Veight" of ttie ttiread. Associating dif- 
ferent weights witfi different real time threads al- 
kyws for favoring one activity over anottter 

In one embodiment of the invention, the step of 
determining when woric is to be done by any of the 
processing threads indudes polling all of the pro- 
cessing tasks. An alternative approach includes en- 
abCng interrupts for higher priority tasks, which 
tmporarily preempt ottier processing. When inter- 
rupts are used, ttie mettiod indudes servteing ttie 
Interrupts as rapidly as possible. Furthermore, ttie 
number of such interrupts alkwed hi any unit of 
time is strictiy controlled, by potentially disabling 
ttie interrupts when a predefined limit of number of 
Intenrupts in an interval Is exceeded. Time consum- 
ing interrupt servidng functions are deferred for 
execution In processing threads, where they must 
compete with the ottier threads for prcx^essing re- 
sources. 

In accordance witti one asped of ttie invention, 
ttie step of dlvicling ttie processing tasks into pro- 
cessing threads indudes establishing real time pro- 



cessing threads, for execution of time-critical func- 
tions, each real time processing ttiread botng struc- 
tured to execute for a time determined by a maxi- 
mum number of processed units; and establishing 

5 at least one general purpose processing thread, for 
execution of func^ons that are less txMjnded in 
terms of execution time. Each general purpose 
processing ttiread is structured to execute for a 
preseleded minimum time, if needed, during which 

10 it is non-preemptable by ottier threads, and to 
execute ttiereafter, If needed, in a preemptable 
mode. During ttie time it is non-preemptat)le, a 
general purpose thread is like a real time ttiread. 
This may be in order to guarantee some process- 

76 ing for general purpose tasks, such as for applica- 
tions, so that they make progress. When a general 
purpose thread has to perform a real time task 
related fonction (e.g., transmit a packet), it may 
queue ttie packet to a real time ttiread. 

20 The step of sdecting a ttiread for execution 
indudes selecting from the threads that have work 
posted, on a round-robin basis. Invoking ttie se- 
lected ttiread indudes switching ttie context of pro- 
cessing from ttie previously executing ttiread to ttie 

25 selected ttiread. and transferring processing control 
to ttie selected thread. Thread selection is made in 
the presentiy preferred embodiment on a round- 
robin basis, le. each ttiread is conskfored for ex- 
ecution In turn, induding ttie general purpose 

30 ttiread or ttveads. tt a ttiread has no woric posted, 
i.e. is not needed at the present time, ttie selection 
process checks the next thread in turn. 

In ttie presentty preferred embodiment of ttie 
invention, ttie step of posting any work to be done 

35 to appropriate processing ttireads Indudes setting 
or posting a flag for each pnx»sslng ttiread ttiat 
has woric to be done. This flag is set when an 
interrupt Is encountered, but no ottier work b done 
for the Interrupt Also, unless ttie flag is reset, ttiat 

40 particular interrupt may not be re-enatried. The 
woric is queued anyway In ttie adapter or driver 
buffers. 

in accordance with another asped of the inven- 
tion, ttie mettiod may also indude limiting the rate 

45 at which interrupts can be servfoed, by temporarily 
disabling interrupts from time to time. This places a* 
limitation on ttie priority kiherent In fntenrupt pro- 
cessing, or lack of control Inherent In priority driven 
interTupt-t>ased processing. 

so TYie invention may also be expressed In terms 
of novd apparatus, for scheduling multiple pro- 
cessing tasks for bi^proved performance in a net- 
work communication station. The apparatus com- 
prises a set of executable processing threads, in- 

65 eluding real time processing threads strudured to 
execute for a seleded maximum time determined 
by counting a seleded number of processed units, 
and at least one general purpose prcx:essing ttiread 
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structured to execute in a non-preemptable mode 
for a selected minimum time, tf needed, and to 
execute thereafter, if needed, in a preemptable 
mode. Two other necessary elements of the ap- 
paratus are a thread posting mechanism, including 
means for repealedty determining when work is to 
be done by any of the processing threads, and 
means for repeatedly posting any woric to be done 
to appropriate ones of the processing tfireads; and 
a thread scheduler, for selecting and invoking pro- 
cessing threads that have work to be done. 

Ihe thread posting mechanism may inchide 
either means for polling all of the processing tasks, 
or an Intenrupt system capable of temporarily pre- 
empting processing when a processing task has 
work to be done. The interrupt system uses mini- 
mal Interrupt sefvk:ing means, for processing inter- 
rupts as rapklly as possible, so ttiat the more time 
consuming intenrupt servicing functions are de- 
ferred for execution In processing threads. 

The thread scheduler of ttie apparatus includes 
means for selecting from ttie real time and general 
purpose threads ttiat have work posted, on a 
round-robin basis; and means for invoking ttie se- 
lected ttiread by switching ttie context of process- 
ing from the prevkxjsly executing thread to ttie 
selected ttiread, and transferring processing control 
to ttie selected ttiread. In ttie Illustrative embodi- 
ment of tfte inventkxt, the means for repeatedly 
posting any work to be done to appropriate pro- 
cessing tfireads Includes means for setting or post- 
ing a flag for each processing ttiread ttiat has work 
to be done. As discussed above, each ttiread has a 
weight associated with It; and ttie weight is used to 
detennine ttie time spent in ttie ttiread before relin- 
quishing control to anottier thread. 

Before entering a real time thread, ttie sched- 
uler automatically clears ttie flag ttiat kKficated 
work needed to be done by ttiat ttiread. The real 
time servtee routine ttiat causes posting of the flag 
indicating work is pending has to check to deter- 
mine if there is work actually pending to be pro- 
cessed, since It is possible ttiat ttie work of ttie 
conresponding intenrupt was already done during 
the previous Invocation of ttie real time thread. 

It will be appreciated from ttie foregoing ttiat 
the present Invention represents a significant ad- 
vance In scheduling mechanisms for use ki com- 
munication networic station processors, including 
host processors and processors in network adapt- 
ers. Ottier aspects of ttie invention will become 
apparent from ttie following more detailed descrip- 
tion, taken In conjunction witfi ttie accompanyirig 
drawings. 



BRIEF DESCRIPTION OF THE DRAWINGS 

RGURE 1 is a block diagram showing a network 
station connected to a communication network 
5 ttirough an adapter; 

RQ. 2 is a block diagram depicting ttie princi- 
ples of the scheduling mechanism of the 
present invention; 

RGS. 2A and 2B depict a real time processing 
10 thread and a general purpose processing thread, 
respectively: 

RG. 3 is a fkm cfiart showing the functions of a 
real time polling thread or a real time processor 
for the same purpose; 
IS RQ. 4 is a ftowchart showing the functions of a 
minimal intenrupt servk^e processing routine; and 
RG. 5 is a ftowchart showing ttie functions of a 
core thread scheduler appearing in RG. 2. 

20 DESCRIPTION OF THE PREFERRED EMBODI- 
MENT 

As shown in the drawings for purposes of il- 
hjstration, ttie present Invention is concerned with 

25 operating system sctieduling mectianisms for use 
in communk:ation network stations. Although the 
medianism of tfie invention is applicat>le to various 
types of network protocols. It will bo described for 
purposes of Olustration in relation to a token ring 

30 networic, such as one following the fiber distributed 
data interface (FDDI) protocols. 

A typtoal network station configuration is shown 
in RQ. 1, and includes a network adapter, indicated 
by reference numeral 10, connected between a 

ds token ring network 12, indicated as l>eing of ttie 
fiber distributed data interface (FDOQ type, and a 
system bus 14. The system Khis 14 has connected 
to It a central processing unit (CPU) 16 and a 
system memory 18. The adapter 10 processes 

40 data packets received from the network 12 and 
stores them in the system memory 18 k>r sut>so- 
quent processing by the CfHJ 16. 

The adapter 10 Inckxles an FDDI interface 20, 
a packet memory 22, a system bus interface 24. a 

45 packet memory controller 26 and an adapter pro- 
cessor 28. Received data packets are stored tem- 
porarily In tfie packet memory 22 and later defiv- 
ered to ttie system memory 18 under control of the 
adapter processor 28. The rate at which received 

50 packets must be accepted from tfie network 12 Is 
beyond ttie control of ttie adapter, and traditionally 
operation of the processor 28 is scheduled on an 
intenrupt driven basis, to ensure ttiat priority is 
given to processing the receive traffic. In this ex- 

55 ample, the CPU 16 performs a file server function, 
which means ttiat it stores and retrieves data in a 
large mass storage device, such as a hard disk 30, 
and performs this function for multiple user stations 
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(not shown) connected to the network 12. 

The prindples of the present invention apply to 
scheduling problems encountered both in the 
adapter processor 28 and in the host CPU 16. The 
network adapter 10 processes receive packets by 
placing them in the system memory 18. Thereafter, 
the CPU 16 is involved in receive packet process- 
ing at various network levels, and is also involved 
in transmit packet processing, eventually placing 
transmit packets in the system memory, for re- 
trieval and transmission by the network adapter 10, 
Moreover, H will be appreciated that the division of 
receh/e processing and transmit processing tasks 
between the adapter 10 and the CPU 16 is one that 
may be nr^de in a number of ways. The adapter 
functions could be perfomned by the CPU 16, but 
in most cases a separate adapter makes more 
sense, since the adapter functions are very much 
dependent on the conununicatkMi channel. The 
CPU 16 in addition may be InvoTved in executing 
file server hinctions. In any event, both the adapter 
processor 28 and the CPU 16 perfomn receive 
processing and transmit processing at some level, 
as well as other types of background processing 
required by the station. Therefore, similar priority 
consideratkins apply both to the adapter processor 
28 and to the CPU 16. 

As discussed in the background section of this 
spedficatton, priority of processing in networic sta- 
tions is typbally determined by a system of hard- 
ware interrupts, the priorities of whk^h are fixed and 
in favor of receive packet processing. In some 
situatkms, it may not be possible to avokl an inter- 
rupt-based solution, because of constraints inv- 
posed by the processor operating system, or for 
other reasons. However, even fat an fatterrupt-t^ased 
scheduHng mechanism, it is possible and desirable 
to Gnut the rate at whkii interrupts are accepted. 
One solution is to size the number of receive 
packet queue entries provkJed at ttie interface be- 
tween the datafink layer and the networic layer. A 
receive Uvetock may then be avoided by disabling 
faiterrupts for a short period of time, during whfch 
processing at higher layers (e.g., the networic layer) 
makes progress and eventually frees up buffers for 
the tower levels (o.g., the datafink layer), so that 
bitenrupts can then be re-enabled. To avoid receive 
tivetocks, to reduce latency, and for fairness in 
allocating processing resources, interrupt process- 
ing is nKxJified to keep the woric done bi Interrupt 
service routines to an extremely snwll amount The 
remainder of servicing woric required for tftat task is 
perfonnned at a tower priority, where ttiere will be 
competition witfi other activity, and tiierefore a 
higher degree of fairness among the competing 
tasks. Even transmit packet processing can com- 
pete more falriy for access to processing resources 
if the interrupt service routines are shortened as 



much as possible. Even with these improvements, 
however, an interrupt-t>ased scheduling mechanism 
still does not guarantee fair access to processing 
resources by all pr(x:essing tasks, unless other 
6 enhancements to the scheduler are implemented. 

An ahemative to interrupt-based scfteduling is 
a polling approach, in which a packet processing 
driver "Gstens" for packets to be transmitted and 
received, and tiierefore can control the amount of 
10 processing at tower layers, as weD as ensure some 
level of fairness between transmisston and recep- 
tion of packets. A disadvantage, is tfiat tfiere may 
be an increased latency in processing a packet, 
due to ttie polling delay. However, the overall per- 
is formance and average latency may be improved 
witti a polling structure, because of the better con- 
trol of processing at the tower levels, parttoularty 
when tfie system is heavily toaded. A properly 
des^ned polling structure can control the amount 

20 of time that each layer or sutilayer executes before 
yiekfing to another task, and can control the maxi- 
mum latency that a received request encounters. 

Andher advantage of polling over an interrupt- 
based mecfianism is encountered in faster sys- 

25 tems. The relative overtiead for interrupt process- 
ing increases as systoms get faster t)ecause, for a 
given Increase fai processor speed, the amount (»f 
time to execute an Interrupt servtoe routine (includ- 
ing hardware later>des, which are often the ones 

30 ttuit do not speed up) does not decrease as much 
as the time to execute a typtoai faistruction se- 
quence. bi addition, if faster RISC (reduced instruc- 
tfon set) macfiines are used, many more registers 
have to be saved and restored when faitonrupts are 

35 processed, so the overtiead in context switching 
increases substantially. 

The embodiment descn'bed here attempts to 
use the features of botti these scheduling ap- 
proaches to provide superior scheduling mecha- 

40 nisms for supporting communications tasks. As de- 
scribed herein, real time processing tasks are 
predefined as being made up of a number of real 
time processing threads." each of which is sched- 
uled as needed and runs to a final or intermediate 

45 completion point witiKxit intenruption or preemption. 
The most Important aspect of a thread is that it is 
defined to run for a limited time before yiekling 
control to a tfiread scheduler, but for pertonmance 
and convenience tfie limited time is not measured 

50 by a timer faiterrupt Rather, each ttuead is defined 
to process a selected number of units of woric. 
such as packets, and^this number of units has boon 
cak:ulated to be equivalent to a desired run time 
tor tiie thread. Calculation of the number of units to 

65 process, or the "weight" of the tfiread. is based on 
throughput requirements relative to tiie other 
threads, and possibly on latency considerations. 
Alternatively, a fine granularity clock could be used 
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to control the thread scheduler, at the expense of 
clock interrupt processing overhead. The schedul- 
ing mechanism has three t»asic components: a set 
of predefined processing threads, a thread posting 
mechanism, which detennines v4iat work Is to t)e 
done and "posts" the work to the appropriate 
thread or threads, and a core thread scheduler, 
which detennines which thread wIN next otrtain 
access to processing resources. 

The thread scheduling mechanism Is shown 
diagrammatically in FIGS. 2, 2A and 2B. As shown 
in HQ, 2, the mechanism b divided into a real time 
domain 40 and a general purpose domain 42. The 
real time domain includes a set of real time pro- 
cessing threads 44, each off whfch perfonms a Bmit- 
ed amount of processing for a certain task, such as 
processing recehre packets between one layer and 
another. Each real time thread 44 Is nor>-intenrup- 
tat)le for the period that H operates, and Its period 
of operatkMi is predetermined by permitting only a 
selected numt)er units to be processed. For exam- 
ple, in processing receive packets a thread may be 
constrained to process a selected number of pack- 
ets before reGnquishing control to a core thread 
scheduler 46. 

Within the real time domain, there Is also a real 
time thread posting mechanism 48, the specifics off 
whfch wiU be further described betow. Ihe hmption 
off the posting mechanism 48 Is to determine 
whether any work needs to be done by the varkMis 
threads, and to "post" ttie work to the appropriate 
threads. Onewaytodothisisfforthe posting 
mechanism 48 to set real time thread fflags 50, 
whfch are used to determine whether the threads 
44 shouM be executed or not. 

FIG. 2A illustrates in general the steps per- 
formed by a real time thread. As shown In bkx:k 
52, the thread is executed to process a preselected 
maximum montier of units, such as packets off 
data. Then, as shown in block 54. control Is yieMed 
to the core thread scheduler 46. When one off the 
threads 44 completes execution because H has 
processed the required maximum number of units, 
the thread sets its own thread flag 50 to ensure 
that a return will be made to continue processing In 
this thread on the next round robin schedule selec- 
tion of this thread. If processing in the thread runs 
to completkxi, and not because the maximum num- 
ber of units has been processed, the thread flag is 
not set, and the thread scheduler transfers pro- 
cessing resources to another thread. 

The core thread scheduler 46 is a special task 
to which control Is transffenred at the end off each 
processing thread. The scheduler Is simply a 
round-robin selection device with knowledge of all 
of the processing threads and access to the real 
time thread flags 50. When one processing thread 
is completed, the scheduler 46 chooses the next 



one in tum having a flag set to Indicate that pro- 
cessing is needed. FIG. 5 shows the basic steps 
perfomned by the core thread scheduler 46. After 
return from a completed processing thread. ir>- 
5 dicated at 56, the scheduler 46 selects the next 
thread that needs to nin (bkx* 58). and yields 
control to the next thread (block 60). The scheduler 
46 Includes at least one time sk)t in its round robin 
system ffor the general purpose domain 42. 

10 The real time domain 40 Includes threads 44 
Involved with packet reception and transmission, 
and some file server tasks, such as processing disk 
storage access requests that can be sab'sfied with- 
out waiting for access to the disk 30 itself. This 

rs assumes that the disk 30 has an associated cache 
menKxy (not shown), in whfch frequently used data 
segments from the disk are stored. In many In- 
stances, requests for reading disk data can be 
satisfied by reading data from the disk cache. 

20 SImilarty. a cache memory can be used for disk 
write requests, and many requests to store data on 
the disk can be satisfied by writing to the catihe 
memory; then later writing the caclie memory to 
tfie disk wtien time is availat>le. Thus, any disk 

25 reading and writing operatfcns only Invoking a 
cache memory can be included as real time 
threads, since they tak^ relatively Gttle time to 
execute. RIe server functions that require disk ac- 
cess, to read data not already in a read cache 

so memory, or to write data to the disk from a wrfte 
cache memory, are performed In the general pur* 
pose domain 42. whfch inckides activities that are, 
in general, "unbounded" in tenms of the time that 
they take to execute. 

35 For fairness off resource allocation, the general 
purpose domain 42 is ^n at least one time slot 
in the rourxf-robln aHocation scfieme used by the 
core thread scffieduler 46. The gerteral purpose 
dom^n 42 is depfcted as having a single thread 

40 62. but H wiD be understood that there may be 
multiple general purpose threads and that the gen- 
eral purpose domain may have to Include its own 
scffieduler to allocate processing time among tftese 
multiple threads. 

45 The general purpose thread 62 is executed for 
a preselected minimum time, as indicated in bfock 
64, and then yiekts control to the core scheduler 
46. as Indicated In bfcck 66. When the general 
purpose thread 62 is initiated, the current minimum 

60 requirements of all of the general purpose domain 
activities are considered, and the minimum thread 
execution time is computed based on these re- 
quirements. After the minimum execution time, tiie 
general purpose thread 62 may continue to mn, bui 

65 may be preempted when any work arrives for a 
real time processing thread. The general purpose 
domain thread 62 may have Its minimum execution 
time determined by a numt}er of units processed. 
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or may use a timer interrupt to end its minimum 
execution time. After the general purpose thread 62 
enters its preemptable stage, preemption may bo 
by way of a hardware intenrupt, or a periodic timer 
intenrupt that checks to determine whether any real 
time taslcs are pending, or by way of polling Ini- 
tiated from the general purpose domain 42 for any 
real time tasks. 

The real time posting mechanism 48 may take 
eltfier of two forms: a polling technk)ue or by a 
limited use of intenrupts. The polling approach re- 
quires ttiat a separate real time tfiread be used to 
check all tt)e station Interfaces for work to be 
performed by the processing ttireads 44. The pol- 
ling tfiread may be given special prk)rity In the 
core thread scheduler 46. The approach Is de- 
pleted in RG. 3, and includes the steps of polfing 
all interfaces for work to be done by the real time 
threads, as shown in bkx* 70, and tiien posting 
any work to the tiureads by setting appropriate real 
time ttvead flags 50. as shown In bk)ck 72. If ttie 
posting mechanism is a processing thread (and not 
a dedk:ated processor), tt>e mechanism tfien yiekis 
control to the tfiread scheduler 46, as shown In 
bkx:k 74. A dedk:ated processor performing ttie 
polling task can merely keep kx>plng tfirough tfie 
steps, as shown by tfie broken Bne 76. 

This polfing approach Is often suitable if ttiere 
is an abundance of processing resources, or if a 
separate processor can be dedk^ted to tfie poHing 
task. In a more resource Gmited singlo-processor 
system, use of a pofiing tfiread to post work to tfie 
real time processing threads may introduce latency 
in tfie reception of a request, especially If there are 
many real time items to be polled and if tfie 
general purpose domain is active. The latency can 
be reduced by polling more often (giving tfie pol- 
fing tfiread higher prk>rity in tfie tfiread scheduler) 
but doing so also Increases overhead usage of tfie 
processing resources. 

A more attractive technk^ue of real time thread 
posting Is presentty preferred, and that Is to make 
Judlckws use of intenrupts. Each real time Interrupt 
performs only the minimum action needed to rec- 
ognize tfie Intenrupt and to post the work to be 
done to an appropriate real time tfiread, as in- 
dicated In bk)ck 80 (FIG. 4). Then tfie intonvpt 
service routine returns control to ttie intenrupted 
processing, as indicated in bkKk 82. Use of an 
Interrupt to record critical real time events, such as 
tfie receipt of a packet, keeps latency tow, since 
there Is no polling delay. Since these interrupt 
service routines perform minimum actions, we do 
not have a tot of context (i.e., registers) to be saved 
and restored. Therefore, the context switching over- 
head has minimal Impact on latency. Once tfie 
interrupt is recognized, processing continues in an 
orderly fashion, using the scheme of processing 



ttireads controlled by tfie core scheduler. 

The execution time or "weight" altocated to 
each real time thread. In terms of the number of 
processed units, depends on two factors. The first 
5 factor is the ttiroughput requirements for the real 
time processing tfiread. For example, a real time 
processing tfiread interfacing a higher bandwidtti 
fink, such as tfie FDDI network, may have a larger 
weight than a real time thread interfacing a stower 

10 speed fink, such as Etfiemet. The second factor 
influencing ttie choice of weights tfie real time 
thread Is the latency requirement of the threads. 
Taking this second factor into account Is, at the 
present stage of development, more elusive. C^lear- 

15 ly. deadfine scheduling for the vartous threads 
should play a role in the overall scftedule, but at 
present the chotoe of thread weights to optimize a 
complex system is stin largely a matter of intuitive 
adjustment of tfiese values. The fofiowing para- 

20 graphs discuss some specific examples of how to 
cak:ulate weights for real time threads. 

If ttie tfireads can be defined to have desired 
throughputs, and latency is not a conskleration, the 
thread weights may t>e calculated as proportional 

2s to tfie tfiroughputs. That Is. tfie weight for ttvead I 
is cateulated as being proportional to tfie desired 
tfiroughput of ttvead I divided ttie sum of ttie 
desired throu^iputs of afi of ttie real time ttireads. 
The desired throughput may be normafized to 

30 some nominal tfiroughput such as that for the 
stowest network fink. For example. If tfiero are 
three tfireads (for Etfiemet driver/jprotocol process- 
ing, FDD! processing, and disk subsystem process- 
tog) witti relative ttiroughputs of 1. 10 and 5, ttien 

3S the weights may be chosen as 1/16, 10/16 and 
5/16. respectively. If ttie scheduler has 16 time 
stots in a fuD cycto of Its operation, the real time 
tfireads will be altocated 1. 10 and 5 time stots, 
respectively. If tfie processing capacity of ttie en- 

40 tire system Is known, for example to packets per 
second, then ttie number of packets processed per 
unit time to each tfiread can also be catoulated. If 
the system capacity Is C packets/second, the 
weights may be chosen proportionately as CI/16, 

45 CIO/16 and C5/16. 

H protocol processing is performed as a com- 
mon tfiread for botfi Etfiemet and FDDI, tfiis would 
add a fourtti tfiread to the example. Assuming ttiat 
tfie protocol processing thread has a relative 

60 tfiroughput of 11 units of tfiroughput produced by 
Etfiemet and FDDI togetfier, ttie weights of ttie four 
tfireads are 1/27, 10/27, 5/27 and 11/27. 

Continuing witti 'tfiis example. It Is assumed 
that tfiere Is a need to meet "soft" real time pro- 

55 cessing requirements (in'tenms of meeting a dead- 
line for executing a task). In particular, if it is 
assumed ttiat there is enough buffering (say 100 
buffers) to avoid excessive dropping of packets 



8 



15 



EP 0 617 361 A2 



16 



because the buffers have filled. Further, it is as- 
sumed in the example that for the Ethemet traffic 
there is a soft deadline of x units of time before 
which an Incoming packet must be sendced. The 
worst-case latency for an Incoming packet from the 
Ethemet can then be cateulated. When a first pack- 
et Is received and the buffers are not occupied, the 
latency for the packet Is 27 time stots. The worst 
case for latency occurs when a packet of interest 
occupies the 100th buffer, and the latency Is 
2ri00 slots. If this latency Is too high, one of two 
design changes may be made: (a) reducing the 
amount of buffering for Ethemet packets so that if 
the guaranteed worst-case latency cannot be met, 
the packet Is dropped, or (b) increasing the relative 
weight for Ethemet processing as connpared to the 
weights of the other threads. The latter change can 
be effected by processing multiple packets for 
each time that the thread is invoked, or, if one is 
Interested In a finer granularity of response time, 
by distributing ttie Ettiemet thread schedule over 
multiple points in the round-robin schedule of the 
thread scheduler. 

The foregoing example has, for simplk:ity, 
omitted any accounting for tiie general purpose 
thread. whk:h occupies an additional time slot in 
fho round-robin schedule of the thread scheduler. 

To some extent, latency Issues can t>e used to 
determine ttie appropriate thread weights. For ex- 
ample. If there are no latency bounds of concern, 
and tite system provkSes a k)t of buffering Ut tho 
kmer layers, then the design couM put all the woric 
of multiple layers, such as ttie datalink layer, the 
networic layer, and pos^lily ttie transport layer, into 
one real time processing ttuead. This would mini- 
mize ttie amount of context switching needed to 
process a single packet On ttie ottier hand, If 
latency is of concern, or H packet toss is a signifi- 
cant Issue, ttie goaS shouki be to have ttie woric of 
processing a packet broken up into multiple real 
time ttveads. The woric of minimizing packet bss 
may be put In ttie first real time ttvead, whk:h Is 
given a weight of multiple units of woric Each unit 
of woric Is a small amount and does not knpact 
latency, but moves ttie packet into an area having 
a relatively large amount of buffering. Processing in 
higher layers is ttien periormed In a different real 
time ttiread. In any event, ttie division of woric into 
multiple ttireads and ttie selection of weights (mini- 
mum non-preemptable processing times) for ttie 
ttireads, is a design process ttiat depends largely 
on ttie available hardware, particularly buffering, 
and on ttie nature of tfie networic and ttie design 
goals, such as minimizing latency and packet k>ss, 
and fairness of processing various tasks. 

Regardless of how these design parameters 
are selected, an Important aspect of the invention 
is that the real time threads, in general, do not run 



to "completion," unlike an interrupt-tesed system. 
In the context of ttiis invention, not running to 
"completion" means eitfier (a) in ttie case of pro- 
cessing a Gst or queue of items, not continuing to 
5 retain control until ttie entire queue is emptied, or 
(b) in ttie case of a single k)ng task, not completing 
ttie task before relinquishing control. Because ttie 
real time ttireads are limited In this manner, ttie 
possibility of a receive Gvelock condition is prac- 

10 tically eliminated. As a consequence, starvation of 
transmit operations Is also a very k>w probability. 
And with appropriate choices of real time threads 
and ttieir weights, packet k>ss and latency can be 
kept to an acceptably k>w level. 

fs Tfie file server used as an Otustratkm in ttiis 
specification poses additional conskSerations. The 
file server may still use disk interrupts in process- 
ing Its cfisk reading and writing activities, in an 
effort to guarantee km latency for ttie file server 

20 functions. However, an of ttie disk Input/output op- 
erations are initiated t)y real time or general pur- 
pose processing ttireads. and it is possible to im- 
pose requirements on tfie cfisk operations, such as 
limiting ttie rate of intenrupts from tfie disk. The 

25 ttireads ttiat schedule disk activity have knowledge 
of ttie file serving func:tion that can bo used to 
control the rate of Intenrupts and to ensure tfiat ttie 
c:riteria of fairness of access to processing re- 
sources are satisfied. 

30 Ttie use of interrupts for processing disk in- 
put/output (I/O) afkms the use of traditional mecha- 
nisms for processing cfisk I/O, whk:h requires kiw 
latency for response to an Interrupt We may not 
have the time as required by a round tdtin scfied- 

35 ule to process a disk interrupt So if we do not want 
to have the disk subsystem take loo much pro- 
cessing rescxJTces and thus cJefeat our fine grain 
control, we must cxmtrol tfie rate of interrupts from 
ttie disk. This Is achieved if an disk l/O's are Issued 

40 from a real time thread such as received packet 
processing (or from a set of real time tfireads). 
These threads can ttten keep an account of how 
much cfisk I/O they have issued and account for the 
time spent in the cfisk interrupt servk» routine 

45 when accounting for ttie weight they assign for the 
real time ttiread. If ttie disk I/O was issued in an 
unsofidted fashion, we will not be able to accxxint 
for tfie processing cycles expended for ttiis woric 
and thus cxir weigfit assignments wouki bo wrong 

50 (because some of the processing cycles are unac- 
counted for). Also disk I/O cannot be put in ttie 
general purpose, domain due to time constraints 
(deadlines). 

A variant of ttie ttiread processing system do- 
ss scrit)ed above uses a different scheme for invoking 
ttie threads. The use of a separate thread for each 
individual activity is very attractive only if the over- 
head for invoking and scheduling a thread is rela- 
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lively small. Each switch to a different thread in- 
volves a switch in processing context and an asso- 
ciated overhead In invoking the new thread. In an 
alternative emtxxliment of the invention, threads 
are not scheduled but are Invoked from real time 5 
service routines. The service routines must ok>6y 
stringent rules to behave in a controlled fashion 
and to establish the necessary context for a newly 
called thread. For example, a thread to process 
receive packets typically begins by receiving a lo 
number of packets, and then completes their pro- 
cessing and deHvers them to be used by another 
routine. Thus tfie routine does not have to recover 
any context, nor save it upon completion, in es- 
sence, the threads are Invoked by procedure-spe- rs 
ctfk: calling routines, to avokJ time consunUng con- 
text switching that involves all context items. 

H will be appreciated from tt^e foregoing ttmt 
fhQ present invention represents a signifteant ad- 
vance In the fiekl of scheduling mechanisms lor 20 
processors used in communicatkm network sta- 
tions. Use of ttie invention provkles for improved 
performance of tha system under normal k>ad, and 
stgnificantiy supertor performance under overk>ad 
conditions. Specifically, ttie schedufing mechanism 2s 
of thB invention practically eliminates receh^e 
Gvekxk conditions and starvation of transmits. 
Moreover, tho technique can be optimized for a 
given system configuration and k>ad comfitions, to 
minimize latertcy and packet loss on reception. It 30 
will also be appreciated tiiat, altixxigh an embodi- 
ment of the invention has been described in detail 
for purposes of illustration, varkMis modifteations 
noay be made without departing from ttie spirft and 
scope of tho invention. 3S 

Claims 

1. A mettiod for scheduling multiple processing 
tasks for improved performance In a n^ork 40 
conununlcation statkm, fh& mettiod comprising 
the steps of: 

divkling processing tasks Into processing 
threads (44), each of whteh is structured to 
execute for a limited time before being subject 45 
to preemption by arK>ther processing thread; 

repeatedly determining (46) when work is 
to be done by any of ttte processing ttveads; 

repeatedly posting (46) any work to be 
done to appropriate ones of tfte processing so 
threads; 

selecting a processing ttiread tfiat has 
work to be done; 

invoking ttie selected processing thread; 

executing (52) tfie selected and invoked ss 
processing tftread for fho limited time, vaierein 
a measure of execution time is obtained in at 
least some of the threads by counting up to a 



selected numt)er of processed units; and . 

repeating tfie selecting (54), Invoking and 
executing steps for ottier processing ttireads. 

2. A metfKxJ as defined in claim 1, wherein the 
step of repeatedly determining when work is to 
be done by any of tfie processing threads 
includes polling (70) all of tfie processing 
tasks. 

3b A metfiod as defined In dalm 1, wherein tfie 
step of repeatedly determining when work is to 
be done by any of the processing threads 
includes: 

enabGng intenupts (80) for higher priority 
tasks, whk:h temporarily preempt otfier pro- 
cessing; and 

senmang intern^ (82) that occur as rap- 
kJly as possible, wherein time consuming inter- 
njpt sefvk:lng functions are deferred for execu- 
tion in processing tfireads. 

4. A metfiod as defined ki claim 1, wherein tt» 
step of dividing the processing tasks into pro- 
cessing tfveads includes: 

estatsRshing real tifne processing threads, 
for execution of time<:ritical functions, each 
real time processing tfiread being structured to 
execute for a time determined by a maximum 
numt>er of processed units; and 

estat>rishing at least one general purpose 
processing thread, for execution of functions 
that are less bounded in terms of execution 
time, wherein the general purpose processing 
thread is structured to execute for a preselec- 
ted minimum time, if needed, during whk:h it is 
non-preemptabfo by otfier threads, and to ex- 
ecute tfiereafter, if needed, in a preemptable 
mode wherein: 

tfie step of selecting a tfiread includes 
selecting from the tfireads tfiat have work post- 
ed, on a round-robin basis; and 

tfie step of invoking tfie selected tfiread 
includes switcfdng tfie context of processing 
from the previously executing tfiread to tfie 
selected tfiread. and transferring processing 
control to tfie selected tfiread, wherein: 

tfie step of selecting a tfiread includes 
selecting from the real time and general pur- 
pose tfireads tfiat have work posted, on a 
round-robin basis; and 

tfie step of ^invoking tfie selected tfiread 
includes switching tfie context of processing 
from the previously executing thread to tfie 
selected tfiread, and transfening processing 
control to tfie selected tfiread. 
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5- A method as defined in claim 1. wherein the 
step of repeatedly posting any work to be 
done to appropriate processing threads In- 
cludes: 

setting a posting flag (50) for each pro- 5 
cesstng thread that has work to be done. 

6. A method for scfieduling multiple processing 
tasks for improved performance In a network 
communtoation station, ttie method comprising 70 
the steps of: 

dividing processing tasks into real time 
processing threads, which are structured to 
execute for a selected maximum time deter- 
mined by counting a selected number of pro- is 
cessed units, and at least one general purpose 
processing thread, whk:h Is struchired to ex- 
ecute in a non-preemptable mode for a se- 
lected minimum time. If needed, and to ex- 
ecute thereafter. If needed. In a preemptable 20 
mode; 

repeatedly determining when work is to be 
done by any of the processing threads: 

repeatedly posting any work to be done to 
appropriate ones of the processing threads; 2s 

selecting a processing thread that has 
work to be done; 

invoking ttie selected processing thread; 

executing the selected and Invoked pro- 
cessing thread; and 30 

repeating the selecting, invoking and ex- 
ecuting steps for ottier processing ttireads. 

7. Apparatus, (10) for scheduling multiple pro- 
cessing tasks for improved performance in a ss 
network communteation station, the apparatus 
comprising: 

a set of executable proces^ng threads 
(44), indudir^ real time processing threads 
structured to execute for a selected maximum 4o 
time determined by counting a selected num- 
ber of processed units, and at least one gen- 
eral purpose (42) processing tfiread structured 
to execute in a noniMreemptable mode tor a 
selected minimum time, if needed, and to ex- 45 
ecute thereafter, if needed, in a preemptable 
mode; 

a thread posting mechanism (48), includ- 
ing means for repeatedly determining when 
woric is to be done by any of tiie processing so 
ttireads, and means for repeatedly posting any 
work to be done to appropriate ones of ttie 
processing ttveads; and 

a ttiread scheduler (46), for selecting and 
invoking processing ttireads that have woric to 65 
be done. 



8l Apparatus as defined in daim 7, wherein the 
ttiread posting mechanism (48) includes: 

means (70) for polling all of ttie processing 
tasks. 

9. Apparatus as defined in daim 7, wherein ttie 
ttiread posting mechanism (48) Includes: 

an interrupt system (80) capable of tonn- 
porarily preempting processing when a pro- 
cessing task has woric to be (tone; and 

minimal bitenrupt servidng means (82), for 
processing interrupts as rapidly as possiblle, 
wherein time consuming interrupt servidng 
functions are deferred for execution in pro- 
cessing ttireads, wherein: 

ttie real time processing ttireads (44) are 
selected for execution of fimeK:rftical functions; 
and 

the at least one general purpose process- 
ing ttiread Is selected for execution of func- 
tions that are less bounded in terms of execu- 
tion time. 

10. Apparatus as defined in daim 7. wherein ttie 
ttiread scheduler (46) inchxtes: 

means for selecting from ttie real time and 
general purpose ttireads ttiat have woric post- 
ed, on a round-robin basis; and 

means for invoking the selected tfiread by 
switching ttie context of processing from ttie 
prevtously executing ttiread to ttie selected 
ttiread, and transferring processing control to 
ttie selected ttiread, wherein the means for 
repeatedly posting any work to be done to 
appropriate processing ttireads inctoctos: 

means for setting a posting flag (50) for 
each processing thread that has work to be 
done. 
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